CLAIMS 



We claim: 

1. A method in a computing system for ranking items in a search result, the 
method comprising: 

receiving a query specifying one or more terms; 

generating a query result identifying a plurality of items satisfying the query; 

and 

for a plurality of items identified in the query result, combining ratings 
reflecting both (a) the frequencies with which users selected the item in query results 
produced for earlier queries specifying one or more terms of the query and (b) levels of effort 
required to make such selections, such that the combination of ratings produces a ranking 
value for the item. 

2. The method of claim 1 wherein the ratings that are combined reflect the 
number of items that preceded the selected items in the query results from which they were 
selected. 

3. The method of claim 1 wherein the ratings that are combined reflect the 
set of navigation commands needed to reach the selected items in the query results from 
which they were selected. 

4. The method of claim 1 wherein the ratings that are combined reflect the 
extent to which the query results from which the items were selected were scrolled to reach 
the selected items. 
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5. The method of claim 1 wherein the ratings that are combined reflect in 
the query results from which the items were selected, the number of pages of each query 
result that preceded the page of each query result containing the selected item. 

6. A method in a computing system for compiling statistics usable to rank 
items in a distinguished query result produced for a distinguished query, the method 
comprising; 

receiving a rating set of queries, each query in the rating set specifying one or 

more terms; 

for each query in the rating set, 

generating a query result identifying one or more items satisfying the 

query; 

allowing a user to select one or more of the items identified in the query 

result; and 

for items selected from the query result, for terms specified by the 

query, 

determining an adjustment factor indicating the level of effort 
necessary to effectuate selection of the item; and 

adjusting a rating score corresponding to the combination of the 
selected item and the term specified by the query by the determined adjustment factor; 

such that rating scores are produced that are usable to rank items 
in a distinguished query result produced for a distinguished query. 

7. The method of claim 6 wherein determining each adjustment factor 
includes increasing the adjustment factor for items occurring in the query result before the 
selected item. 

8. The method of claim 6 wherein determining each adjustment factor 
includes increasing the adjustment factor for navigation operations performed to reach the 
selected item in the query result. 
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9. A computer memory containing a user behavior data structure usable to 
rank the relevance of items in a query result, the data structure comprising a plurality of 
rating scores, each rating score corresponding both to a query term and to an item, and 
reflecting quantitatively the extent to which users have selected the item from query results 
generated from queries specifying the query term and the level of effort required to make 
such selections, 

such that the data structure may be used to rank items in a distinguished query result 
produced for a distinguished query by, for each item in the distinguished query result, 
retrieving from the data structure the rating scores corresponding to the item and any term 
specified in the distinguished query and combining the retrieved rating scores to generate a 
ranking value for the item. 

10. A method in a computing system for assessing the usefulness of a 
distinguished item relative to queries containing a distinguished query term, comprising: 

from among a plurality of executed queries, identifying queries containing the 
distinguished query term in whose query results the distinguish item was both (a) included 
and (b) selected; 

for each identified query, determining a constituent score reflecting the amount 
of effort needed to select the query; and 

combining the determined constituent score to obtain an overall score 
indicating the usefulness of the distinguished item relative to queries containing the 
distinguished query term. 

11. The method of claim 10, further comprising using the obtained overall 
score to position the distinguished item in the query result of a query containing the 
distinguished term. 

12. The method of claim 10 wherein the determination of each constituent 
score is based on a position of the distinguished item within the query result of the identified 
query. 
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13. The method of claim 10 wherein the determination of each constituent 
score is based on a level of prominence of the distinguished item within the query result of 
the identified query. 

14. The method of claim 10 wherein the determination of the constituent 
score for each identified query is based on the navigation commands issued after the query 
result is displayed and before the distinguished item is selected. 

15. The method of claim 10 wherein the determination of the constituent 
score for each identified query is based on an amount of time that elapses between the 
display of the query result and the selection of the distinguished item. 

16. A computer-readable medium whose contents cause a computing system 
to rate the usefulness of a distinguished item relative to queries containing a distinguished 
query term by: 

from among a plurality of executed queries, identifying queries containing the 
distinguished query term in whose query results the distinguish item was both (a) included 
and (b) selected; 

for each identified query, determining a constituent score reflecting the amount 
of effort needed to select the query; and 

combining the determined constituent score to obtain an overall score 
indicating the usefulness of the distinguished item relative to queries containing the 
distinguished query term. 

17. A computing system for assessing the usefulness of a distinguished item 
relative to queries containing a distinguished query term, comprising: 

a query identification subsystem that identifies from among a plurality of 
executed queries queries containing the distinguished query term in whose query results the 
distinguish item was both (a) included and (b) selected; 

a constituent score determination subsystem that determines a constituent score 
for each identified query reflecting the amount of effort needed to select the query; and 
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an overall score determination subsystem that combines the determined constituent score to 
obtain an overall score indicating the usefulness of the distinguished item relative to queries 
containing the distinguished query term. 



18. A method in a computing system ranking items in a search result, the 
method comprising: 

receiving a query specifying one or more terms; 

generating a query result identifying a plurality of items satisfying the query; 

and 

for a plurality of items identified in the query result, combining ratings of 
frequencies with which users selected the item in earlier queries specifying either (a) one or 
more terms of the query or (b) a term sharing a root with a term of the query to produce a 
ranking value for the item. 

19. A computer-readable medium whose contents cause a computing system 
to rank items in a search result by: 

receiving a query specifying one or more terms; 

generating a query result identifying a plurality of items satisfying the query; 

and 

for a plurality of items identified in the query result, combining ratings of 
frequencies with which users selected the item in earlier queries specifying either (a) one or 
more terms of the query or (b) a term sharing a root with a term of the query to produce a 
ranking value for the item 

20. A method in a computing system for compiling statistics usable to rank 
items in a distinguished query result produced for a distinguished query, the method 
comprising: 

receiving a rating set of queries, each query in the rating set specifying one or 

more terms; 

for each query in the rating set, 

generating a query result identifying one or more items satisfying the 

query; 
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allowing a user to select one or more of the items identified in the query 

result; and 

for items selected from the query result, for terms specified by the 
query, adjusting a rating score corresponding to the combination of the selected item and the 
root of the term specified by the query, the rating score indicating the relative frequency with 
which users have selected the selected item when the selected item has been identified in 
search results generated from queries containing a search term sharing a root of the search 
term specified by the query, to produce rating scores usable to rank items in a distinguished 
query result produced for a distinguished query. 
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